A Stochastic Trust-Region Framework for Policy Optimization

In this paper, we study a few challenging theoretical and numerical issues on the well known trust region policy optimization for deep reinforcement learning. The goal is to find that maximizes total expected reward when agent acts according policy. subproblem constructed with surrogate function coherent general distance constraint around latest We solve using preconditioned stochastic gradient method line search scheme ensure each step promotes model stays in region. To overcome bias caused by sampling estimations under random settings, add empirical standard deviation of predicted increase ratio order update radius decide whether trial point accepted. Moreover, Gaussian which commonly used continuous action space, maximization respect mean covariance performed separately control entropy loss. Our analysis shows deterministic version proposed algorithm tends generate monotonic improvement global convergence guaranteed moderate assumptions. Comparisons state-of-the-art methods demonstrate effectiveness robustness our over robotic controls game playings from OpenAI Gym.